fix(profiling): invalidate stale Echion frames on Python 3.14 by taegyunkim · Pull Request #19641 · DataDog/dd-trace-py

taegyunkim · 2026-08-11T22:46:06Z

Description

On Python 3.14, Echion's frame cache can return stale metadata when CPython reuses a released code object's address. Before each sampling cycle, snapshot every interpreter's (interpreter ID, code object generation). If that snapshot changes, clear the global frame cache and the cached asyncio and uvloop frame identities before unwinding stacks.

If the interpreter snapshot is incomplete, clear the same state and skip that sampling cycle. This preserves Echion's existing multi-interpreter traversal, including processes that host a legacy subinterpreter without importing ddtrace there.

Python 3.13 and earlier are unchanged because they do not expose _code_object_generation.

Testing

Added a native test covering unchanged, reordered, changed, replaced, and incomplete interpreter snapshots.
Added a focused Python 3.14 regression test that forces code-object address reuse and verifies that the replacement frame is reported without stale metadata.
Verified the focused regression passes 5/5 runs with the fix and fails 5/5 runs with generation invalidation disabled.
Merged main at 6f2816b8e and rebuilt the Python 3.14 native extension with warnings as errors.
Ran the focused address-reuse regression and the post-fork span-link integration test after resetting the sampler scratch vectors.
Ran clang-format 18.1.5, Ruff 0.14.10, codespell, git diff --check, and the profiling native coverage check.

Risks

Workloads that frequently create code objects will clear Echion's frame cache more often. The invalidation is limited to Python 3.14 and keeps the existing cache structure and sampling behavior otherwise unchanged.

An unreadable or inconsistent interpreter snapshot skips one cycle rather than risking stale frame attribution.

The generation snapshot is intentionally lock-free and taken once per sampling cycle. Code-object deletion after that snapshot can leave frame metadata stale for the remainder of the current cycle; the next cycle observes the generation change and clears the cache. Eliminating this bounded race would require additional per-frame synchronization or identity tracking that is outside this change.

Additional Notes

This intentionally does not add per-interpreter frame caches, change frame-cache keys, or add code watchers for older Python versions.

Pre-review checklist

The changes have been tested in real services in staging
The PR does not result in new crashes

datadog-datadog-prod-us1 · 2026-08-11T22:50:20Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 7340e29 | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-08-11T23:12:53Z

Benchmarks

Benchmark execution time: 2026-08-11 23:12:51

Comparing candidate commit 93e735b in PR branch taegyun/prof-echion-code-generation with baseline commit 81f65c3 in branch main.

Found 0 performance improvements and 6 performance regressions! Performance is the same for 606 metrics, 10 unstable metrics.

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.319µs; +2.527µs] or [+11.061%; +12.052%]

scenario:iastaspects-add_aspect

🟥 execution_time [+7.633µs; +9.227µs] or [+7.444%; +9.000%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+93.075µs; +99.263µs] or [+23.152%; +24.692%]

scenario:span-start

🟥 execution_time [+1.283ms; +1.443ms] or [+8.023%; +9.019%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+287.233ns; +327.849ns] or [+10.933%; +12.479%]

scenario:tracer-small

🟥 execution_time [+30.068µs; +32.669µs] or [+8.857%; +9.623%]

cit-pr-commenter-54b7da · 2026-08-12T00:38:21Z

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

cit-pr-commenter-54b7da · 2026-08-12T00:38:25Z

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 255 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 255 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.google_adk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.debugging._exception.replay -×-> ddtrace.trace  (product:debugging -> product:tracing, score=132)
ddtrace.debugging._debugger -×-> ddtrace.trace  (product:debugging -> product:tracing, score=132)
ddtrace.llmobs._integrations.openai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

cit-pr-commenter-54b7da · 2026-08-12T00:39:17Z

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/datadog/profiling/stack/echion/echion/echion_sampler.h  @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/echion/echion/interp.h         @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/fuzz/fuzz_echion_interp.cpp    @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/include/sampler.hpp            @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/echion/interp.cc           @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/sampler.cpp                @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/test/test_sampling_cycle_state.cpp  @DataDog/profiling-python
releasenotes/notes/profiling-echion-code-generation-6df0c11c31a5d70c.yaml  @DataDog/apm-python
tests/profiling/collector/test_stack.py                                 @DataDog/profiling-python

…ode-generation # Conflicts: # ddtrace/internal/datadog/profiling/stack/test/test_sampling_cycle_state.cpp

taegyunkim · 2026-08-14T14:10:41Z

@codex review

Copilot

Pull request overview

Prevents stale Echion frame attribution on Python 3.14 by tracking interpreter code-object generations and invalidating frame identity caches.

Changes:

Captures interpreter generation snapshots before stack sampling.
Clears frame and asyncio/uvloop identity caches when generations change or snapshots are incomplete.
Adds native and Python regression coverage plus release notes.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`tests/profiling/collector/test_stack.py`	Tests code-object address reuse.
`releasenotes/notes/profiling-echion-code-generation-6df0c11c31a5d70c.yaml`	Documents the fix.
`ddtrace/internal/datadog/profiling/stack/test/test_sampling_cycle_state.cpp`	Tests generation-based invalidation.
`ddtrace/internal/datadog/profiling/stack/src/sampler.cpp`	Captures snapshots before unwinding.
`ddtrace/internal/datadog/profiling/stack/src/echion/interp.cc`	Reads generations and reports snapshot completeness.
`ddtrace/internal/datadog/profiling/stack/include/sampler.hpp`	Adds interpreter snapshot storage.
`ddtrace/internal/datadog/profiling/stack/fuzz/fuzz_echion_interp.cpp`	Adapts the fuzz harness to the new return value.
`ddtrace/internal/datadog/profiling/stack/echion/echion/interp.h`	Extends interpreter metadata and traversal API.
`ddtrace/internal/datadog/profiling/stack/echion/echion/echion_sampler.h`	Implements cache invalidation and fork reset behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce98e9870e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

taegyunkim · 2026-08-14T15:42:58Z

@codex review

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tests/profiling/collector/test_stack.py:193

The regression does not prove that the old function was sampled before its code object is released. If the sampler misses this first execution window but samples the replacement, both final assertions pass even with generation invalidation disabled because no stale cache entry was ever created. Inspect the profile emitted here and require old_name to be present before proceeding.

        ddup.upload()

chatgpt-codex-connector · 2026-08-14T15:46:39Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 7340e2991f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

fix(profiling): invalidate stale Echion frames on Python 3.14

1040790

taegyunkim force-pushed the taegyun/prof-echion-code-generation branch from 73b02b6 to 1040790 Compare August 13, 2026 18:54

taegyunkim changed the title ~~fix(profiling): invalidate Echion frames after code deletion~~ fix(profiling): invalidate stale Echion frames on Python 3.14 Aug 13, 2026

taegyunkim added 2 commits August 13, 2026 19:00

Merge remote-tracking branch 'origin/main' into taegyun/prof-echion-c…

b69f8b5

…ode-generation # Conflicts: # ddtrace/internal/datadog/profiling/stack/test/test_sampling_cycle_state.cpp

test(profiling): focus Echion address reuse regression

ce98e98

taegyunkim requested a balanced review from Copilot August 14, 2026 14:10

Copilot started reviewing on behalf of taegyunkim August 14, 2026 14:11 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Comment thread ddtrace/internal/datadog/profiling/stack/include/sampler.hpp

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread ddtrace/internal/datadog/profiling/stack/src/sampler.cpp

Comment thread ddtrace/internal/datadog/profiling/stack/include/sampler.hpp

fix(profiling): reset sampler scratch vectors after fork

7340e29

taegyunkim requested a balanced review from Copilot August 14, 2026 15:42

Copilot started reviewing on behalf of taegyunkim August 14, 2026 15:43 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(profiling): invalidate stale Echion frames on Python 3.14 - #19641

fix(profiling): invalidate stale Echion frames on Python 3.14#19641
taegyunkim wants to merge 4 commits into
mainfrom
taegyun/prof-echion-code-generation

taegyunkim commented Aug 11, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Aug 11, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

taegyunkim commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

taegyunkim commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

taegyunkim commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Risks

Additional Notes

Pre-review checklist

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Aug 11, 2026

Benchmarks

scenario:httppropagationinject-ids_only

scenario:iastaspects-add_aspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:span-start

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:tracer-small

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

⚠️ Existing dependency direction violations

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

taegyunkim commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

taegyunkim commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

taegyunkim commented Aug 11, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Aug 11, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 •

edited

Loading